{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JPdQvYmlWmNc"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/llama-2/llama-2-70b-chat-agent.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/llama-2/llama-2-70b-chat-agent.ipynb)\n",
        "\n",
        "# LLaMa 70B Chatbot in Hugging Face and LangChain\n",
        "\n",
        "In this notebook we'll explore how we can use the open source **Llama-70b-chat** model in both Hugging Face transformers and LangChain.\n",
        "At the time of writing, you must first request access to Llama 2 models via [this form]() (access is typically granted within a few hours).\n",
        "\n",
        "---\n",
        "\n",
        "\ud83d\udea8 _Note that running this on CPU is practically impossible. It will take a very long time. If running on Google Colab you go to **Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > A100**. Using this notebook requires ~38GB of GPU RAM._\n",
        "\n",
        "---\n",
        "\n",
        "We start by doing a `pip install` of all required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K_fRq0BSGMBk",
        "outputId": "94aaa24d-53c2-4fc0-f89f-4902a65a8123"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m56.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m244.2/244.2 kB\u001b[0m \u001b[31m28.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m79.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m109.1/109.1 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m92.5/92.5 MB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m32.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m88.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m81.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -qU \\\n",
        "    transformers==4.31.0 \\\n",
        "    accelerate==0.21.0 \\\n",
        "    einops==0.6.1 \\\n",
        "    langchain==0.0.240 \\\n",
        "    xformers==0.0.20 \\\n",
        "    bitsandbytes==0.41.0"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "VHQwEeW9Zps2"
      },
      "source": [
        "## Initializing the Hugging Face Pipeline"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "mElf068NXout"
      },
      "source": [
        "The first thing we need to do is initialize a `text-generation` pipeline with Hugging Face transformers. The Pipeline requires three things that we must initialize first, those are:\n",
        "\n",
        "* A LLM, in this case it will be `meta-llama/Llama-2-70b-chat-hf`.\n",
        "\n",
        "* The respective tokenizer for the model.\n",
        "\n",
        "We'll explain these as we get to them, let's begin with our model.\n",
        "\n",
        "We initialize the model and move it to our CUDA-enabled GPU. Using Colab this can take 5-10 minutes to download and initialize the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 730,
          "referenced_widgets": [
            "36b1cf4e24814e80ac4dc6f0b67411b6",
            "915efb0a863244c0a4b79c7977caea9d",
            "94495f44f0a54c7783eb38e9694fee9f",
            "906ae44dc3d74e10af1104e6313da08f",
            "6d68d15c272e4695ad8fd92e75c13a9c",
            "b46ac6b0ab074ddea1f26c314f74b1d7",
            "aebe3f88b309481a8a6d31908a65e371",
            "682f48978e544bf3aea000d3053e3415",
            "6d51920505484b12a5855a5caba99ef7",
            "0bbcc38f7ef14b14a076d8fb56920132",
            "3e8ee3014ff9415985ed96ac26a59c46",
            "4a004446a3344ba2a7091d06807ad213",
            "0bbfde73ee5142f4ac2ee7c89934727b",
            "8f203e1af3294fc4a9dffb6ad6115eda",
            "5c900f2e1d0246d29213c0004e89cac2",
            "335c947d79da4c4da2c2f0d51cca69ee",
            "ef51c0c4e457442ba2ab514f2e9baca1",
            "723af5c92a8c46b7adf45a1ac2fd26af",
            "da61590cccd14062bade595e0405437d",
            "24dce52d691146efa9a5f5d372722256",
            "4c4ee7c4d7a1484c91f31674277d4b04",
            "426772abcbf643c18d41f6f90c12454c",
            "a733f1f20b0e45419d7889e9a5393fed",
            "959c7dff628b4742929b6a977077fccd",
            "49aa383102b948df8ebfdaad47741207",
            "1a5229800b274a6dbea75c9a09e3f7d5",
            "78eaca928ddd46168b131e3ee1f5a8e9",
            "357540398dad4c1588379fcbc98d9ddf",
            "5493f1027fe44d0ca18c31cdcbf6374f",
            "176418bc91c645d38aab23f98b82ecb5",
            "b115d35568b94e3697450db33ac4d153",
            "9217323dc4da4b91b27fb0f188849885",
            "2adb9c8fcd9f4f628f4543edd71ea390",
            "8fcfd788573642e9b6ec5940446d9786",
            "7fc1fe203ee744698b23a8db440fb941",
            "c19d311d1be44522ba8dadfee2871a88",
            "fa38bfd4e1284513945c3bec03404e57",
            "96463be65e0a4a148c098bf6dfc050a2",
            "ef6c38bc3426429ab8f1ba0a5940d152",
            "16a853036f1c46e48d655743567587ff",
            "790cc60a32e04e9fa7b5ee03ac132487",
            "45067dc2112d469d813c58e77638bebe",
            "7ed2a2525e3b4bcea560be5ca1691b26",
            "bd8b0c855b324859a5ba6a8d8e0b8848",
            "0ef61d1a3c0d49a6bb3fc05f00503822",
            "fd65f8da83814d5787d4b93807c0c4de",
            "15711363aebb4ec2ad14bf5abeb37d99",
            "349b7eae677340b3ab0a2d9a871f5aaa",
            "b8c53437ad924c0b8fd931bd70be98b8",
            "14ea92ec7b3f4fc79de5f46d4247c727",
            "21717201c3a34c1ca9558935d7cdab2f",
            "0b76f88c94f44072a9f03c65a28b5047",
            "3d606cc4c1ba4dce83a88b935773199d",
            "dc3fe7705baa4b33a828ba3db185eac6",
            "db3b20f0796548b6b2529762752dcaf4",
            "e6db73a5e5db4317b32b1e84388ad5b6",
            "f1be59de94574eaf8d3a5f6f522aa31e",
            "0d2b2e8bf0f84024942187ee634dcc4d",
            "229dbefaf55a411d8214a1779f0b1494",
            "41e8900f2cd141e3870a82f798faad09",
            "80c9943cef114b2c8e5287cb53474510",
            "664a912529074b4ab5035a25fb099bf6",
            "41d7e53b19054204a075e696c55181c1",
            "3325330921a142a5af0fafff7e149411",
            "a4aec128a2b2473894131b07a9eef7eb",
            "531e45726e2341b1a308693711870db3",
            "4e457951051540a3b7deaf1fde0ca748",
            "b60ea085a6834f6baf4a64cafb6c5e13",
            "11a3cf9575794ccea5e5232d69246c69",
            "1450ad300b7945d0af3c4462656910d1",
            "06e2f0915d084de39cf9d941557eedae",
            "6e98cd8e102b498183fa68d6bbe20ca1",
            "8a7deba8341f4bcd87172abb767dbabd",
            "ebed4a65db5c4a81b92f2011551f63d2",
            "245af8902e1f4581be1f839ea576f72f",
            "86cf258bae2143d3a6e9403711a6bfe2",
            "b6988ef87e794c9690847936684ed664",
            "bcbdfa36e8cb4cfa82ca95986549793c",
            "c7727e76d0d64eecafe3969e6101018d",
            "585dae9cd4d743b8901bf67b913844a3",
            "8465b8a706eb40f78d8a2dde21b6fbd8",
            "284cca73393141bcaec1d51adaf99707",
            "d58f6f0bae6a4eb7af90ed0e3bbdc88c",
            "a24ae133e4114c73842603b24004917a",
            "1108f9859471468dac5bbe72af954b20",
            "43e691634f864f76b5fd23504d6daddd",
            "ce61ce7cda8f4685a7e1535c799eed80",
            "93884d99b44f4b15b130ad0be787b2f2",
            "a2e872328d8446d2b3a5084376dea438",
            "ada9de8795dc4f4a92772e499339a4f7",
            "34c02a67e57f4c0aa07583bacf1b6f15",
            "0ea07ad1b1bb44c1a8938fff47c17f57",
            "efba83b013fd47a3ae2cdcf0f248b7aa",
            "8cc721c064ea46c593035cd97f935101",
            "7e296aa22b5c48e89e0381255bae3b4c",
            "f477e6b8c2604874ab28fae5b99b8727",
            "d37922b357d5408f92abc5c45771ac64",
            "f76c9e100f3d41b7bb0410d0ed7b4231",
            "1e25c67e535141c0897ac4c72de323e6",
            "7f783a59e79a4834a26a1d4726cc2144",
            "af5e91394f684183b0d3ae97f3c692df",
            "8065adcca363448398797635328a46ed",
            "8eaa575b93bc4436912ff95abd55cf7a",
            "d7ce068841804e5a9348a7dbc3e2156a",
            "1eab7f4fb74a4537bc995941eeee3051",
            "13d07429a2ca407fbe4b356ee2813c1f",
            "9d83e55455d640fba13e6ce70ecca26c",
            "38dc1af25a0641fc8e6efa44792b3ba3",
            "83c246294828450ea27902ff9b4f8960",
            "d515e38462f6419880699881d238668c",
            "96b3f664fb1e450ca0119ad0a5690435",
            "9cb77b3f831a4c29a8e660f104e6186a",
            "d7ffce35736547de86c75f9d2579343c",
            "5f0d6616328342a8a29a6f5ba870d76f",
            "c1baee4491be4b45963feab564344e5b",
            "17404a80e8404fff9252aa19881cccfc",
            "7d28c361d07849c2bd39c1798c703c07",
            "03dbb5dfdb6e4199b890640896ff8603",
            "1ae5445475204f0d9a7e994241a537a2",
            "4d9e659dc9e1450bbe512c55e2de979c",
            "58bff4c119d947118589bc9e0ca9c6e6",
            "33880b111ede48ffb8b2a36661c37adc",
            "d3266a9eabdb4efd854aa043b8e401d7",
            "6184d5feec09452e8995ca701965e89b",
            "43b9bee8fd6a4accaea4b83bcb2057c3",
            "dd24d892e77c4552b7bbebbe7857cf30",
            "0adde746130c48968e00b77c42387188",
            "9de373594ae448548f30693992fe6624",
            "3e4eb0d268f541dd910b795aadbb819f",
            "597ce91c73f7444486e2a8ab30c3ad2a",
            "56b443c208a0491eba6ddd5618f740c1",
            "187039b59b6747929ccc139f68972f16",
            "7a020fba56134d398f1e5f42f261ed6d",
            "040da2178e0d40609d2fec3bbb7afb6b",
            "63c18f7812414d3e93abe2cc4ba151c8",
            "3cb0678730cc4cf18dfd278bbe764121",
            "5daec7407481458f9d9a31d5f3e02090",
            "42d97ed149714cb5807354924430cfd3",
            "3037beb2f31e4995bdeaef1cc0033e44",
            "791a003671cb491b90ba3d6fb5aba03e",
            "e2e25836104d4cec81657f341cf44c6a",
            "cf57b458dcb943dc9e8c4504a2daeda1",
            "1ef79932456d45448f63756ecd9987be",
            "10872141dcf94416856e5c4f4da62e41",
            "06f55fd1b03749a4ab4c1c1e71c9e278",
            "c879f2bd785c4c0f90cb878353dd6e03",
            "463093565d5a457fb6299daf4bf16ae7",
            "ea03e5884261469dae943c0f86a185ba",
            "6f484d4a454749a2809cf9ead753fb31",
            "c4527cc698404eb092ea0191d8083ef9",
            "3ee8451c3c4b480a97046ddb3edf8221",
            "a64df8b1daa14996b644664ca77314be",
            "a478dc2312f84c959a1d41d9c2039145",
            "36cccb49a0a74974951051fe0d3320dc",
            "076fd7659cea4294bdf60fbe331d2a97",
            "6c3ae825bc89443fbc740fba1c7111a2",
            "e29d5c360db94bccbb0ef81ca46b38ea",
            "cdc2455e46714bcca5c9c19326e2c498",
            "7deae743b3074819bf9e53479549d385",
            "23c93eb867b24110b7ed6cf0e659d24c",
            "64f283c262874697b7c8cf09723e88fd",
            "c724757f5c254115bfc917b6e959255f",
            "b66bd62475c64e94a56eba3a235fbef6",
            "d9948f6633cf4f279b5b3b920f097e6c",
            "be09a6f8754e46c1b0627445ff7bff7e",
            "e873901d00df4c90a96953047ae0e51c",
            "f5740182187c42fe858ab4421c04fdb3",
            "da02dd1be4f944e2989bb76bc7f7a668",
            "17ba23c504b94ec2bf26fc94602289b4",
            "d483c07c1b6148959c752d7d8454415e",
            "5d72df3780b345f3a9c1ca54d393d7dd",
            "8c17e72dbbae45a6aaf1cac963bedc80",
            "66bdd8762c604cc9b80fdf0c2992d535",
            "32032395537e4cea95b328f53f6bff0c",
            "57e00ebc31e74ff8ad67cef484b16219",
            "84e5d8e248434516bef89f9876dff2f4",
            "7127122a3b60429fa1f0fa2ea7c29b5e",
            "febaae93da0e455495a78e28f23c784f",
            "54affcfd82014d93aee5bb9d1033d7ca",
            "9932a3f661f843729cb0a7edaaa87a5c",
            "567129eb4b164c15bbccb35ef0f66069",
            "508aa186aaf241a0b6f7aae8649515e4",
            "a9f28f207ae146598f077b4d9b87aceb",
            "12a17190958d422cb3d5b2d187ad2bfd",
            "90c30cf9eaf84aa1bb83ec2dcb023872",
            "cba57de7260444aab8687a40159a5e1d",
            "bbf9d167e20b4b02a217a17194724bd6",
            "e6782f9e00dd4f999466ab241ca6b9c8",
            "7aef5421177f45c0a710a9abc8182690",
            "dfb8b3ce81924ae1b77c8de5b13861f1",
            "022ef33e96af4cd5aee5488f66a6d50e",
            "3a99c0f3601240d4af478e1090ed4341",
            "68516a80341543598c44e32ee488aeee",
            "0993b92b269143aba550d0131894d249",
            "5d9f06fee146445588a6f4b7fbcb8784",
            "61eeb50a72ed4291b28d00ef6598fbde",
            "9223e970f4674fecbca733c1a5b3ad4f",
            "23a1190d4f7240c38c34a1cd664fa723",
            "43dc89eb9cf84f248c80b379707229b6",
            "5c76a2a11bf049d8bbdcb6093eafea3a",
            "971650a031e64dec9c9f0239feab6203",
            "753cd63c2a314b30b92ea56c8ab6c7dc",
            "23f0dba22df1488793452dde09feb1e8",
            "87a4d607cee54f96aa431ca1923fb7a0",
            "03d6e330da2e4791a1622b1d10004d8e",
            "87a69e4fcf70498ab71ae00c4b2cee2c",
            "d3ff8acef6b64f349087809655a6f470",
            "f6f78e623a4641178a8de05ca97bd67a",
            "cbbb16c080d640aeb1427be3b41cc851",
            "a754ab8aa3784badba9b15d936692ad8",
            "0c5672c6b2ad4943b74c87d166f464c7",
            "e4df8bcab8104fb9a88f92ccc8fd54f8",
            "27be12dd71074f54ae7991b95d2d6f7c",
            "9aa62d36449641a8bb7a7c4f8db3ca2c",
            "fb3fb9c0ed454c1b9fbd5eb4df7871f0",
            "995b416feede40ee9945d30c2b46fd09",
            "d1fd833944f64b39a1ad8d4cca830046",
            "ead6038247b34fc1b6a4fa7746a0fffd",
            "91921c79f8d74e65a324ca46794f85f0",
            "5515fd5b4d724f06a030dc324f15155a"
          ]
        },
        "id": "ikzdi_uMI7B-",
        "outputId": "d4ded4bf-5cab-4e25-d231-d981a90dcece"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "36b1cf4e24814e80ac4dc6f0b67411b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)lve/main/config.json:   0%|          | 0.00/635 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:2193: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4a004446a3344ba2a7091d06807ad213",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)fetensors.index.json:   0%|          | 0.00/66.7k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a733f1f20b0e45419d7889e9a5393fed",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading shards:   0%|          | 0/15 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8fcfd788573642e9b6ec5940446d9786",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.85G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0ef61d1a3c0d49a6bb3fc05f00503822",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e6db73a5e5db4317b32b1e84388ad5b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4e457951051540a3b7deaf1fde0ca748",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bcbdfa36e8cb4cfa82ca95986549793c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a2e872328d8446d2b3a5084376dea438",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7f783a59e79a4834a26a1d4726cc2144",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "96b3f664fb1e450ca0119ad0a5690435",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "33880b111ede48ffb8b2a36661c37adc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7a020fba56134d398f1e5f42f261ed6d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "10872141dcf94416856e5c4f4da62e41",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "076fd7659cea4294bdf60fbe331d2a97",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e873901d00df4c90a96953047ae0e51c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7127122a3b60429fa1f0fa2ea7c29b5e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/9.50G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e6782f9e00dd4f999466ab241ca6b9c8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00015.safetensors:   0%|          | 0.00/524M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "43dc89eb9cf84f248c80b379707229b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/15 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a754ab8aa3784badba9b15d936692ad8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)neration_config.json:   0%|          | 0.00/175 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model loaded on cuda:0\n"
          ]
        }
      ],
      "source": [
        "from torch import cuda, bfloat16\n",
        "import transformers\n",
        "\n",
        "model_id = 'meta-llama/Llama-2-70b-chat-hf'\n",
        "\n",
        "device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'\n",
        "\n",
        "# set quantization configuration to load large model with less GPU memory\n",
        "# this requires the `bitsandbytes` library\n",
        "bnb_config = transformers.BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type='nf4',\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_compute_dtype=bfloat16\n",
        ")\n",
        "\n",
        "# begin initializing HF items, need auth token for these\n",
        "hf_auth = '<YOUR_API_KEY>'\n",
        "model_config = transformers.AutoConfig.from_pretrained(\n",
        "    model_id,\n",
        "    use_auth_token=hf_auth\n",
        ")\n",
        "\n",
        "model = transformers.AutoModelForCausalLM.from_pretrained(\n",
        "    model_id,\n",
        "    trust_remote_code=True,\n",
        "    config=model_config,\n",
        "    quantization_config=bnb_config,\n",
        "    device_map='auto',\n",
        "    use_auth_token=hf_auth\n",
        ")\n",
        "model.eval()\n",
        "print(f\"Model loaded on {device}\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JzX9LqWSX9ot"
      },
      "source": [
        "The pipeline requires a tokenizer which handles the translation of human readable plaintext to LLM readable token IDs. The Llama 2 70B models were trained using the Llama 2 70B tokenizer, which we initialize like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 201,
          "referenced_widgets": [
            "35c1a1e0fa5442cc9b070ac887fd40ab",
            "3b04c5769a2846609ce00796ff9a35a5",
            "3b1617add6f247bfbac61de46f0b03aa",
            "27136f0a471f4a65ab282b2e9cd41432",
            "91065a8af80043d2a1355e18701e9587",
            "6cd0e95126ed4d69983463340d842ea5",
            "7b0a63ea3eed4d509aeca380436e7b3b",
            "6a2e320546624e32b8ef0ba443f9195f",
            "192678bc21c343d591f1d7d5c31add66",
            "054381d15586471cbb6c0dba2c964e3c",
            "9ebef01ebe1b4382850d3e3b2a18991a",
            "772e95b70424423b80fbd0f1ffb7e300",
            "9c2d9b9d5da74c4fa34beae0fee9f36b",
            "7d6a9c85c47948f9b5b3630b9d0e07c8",
            "91ba5acc92ff4b5e91b85fa688227178",
            "ca7fcf242d814f8bad230f0edeb9c77f",
            "5badc639d5a14cf7ab8297779f387c40",
            "2ef8c9a4401b4bd2af456330feb5aed6",
            "4c502d3b39d34c80b36388a9d73c8949",
            "dacb68c54b894a2f88cba718e9385583",
            "97a53312ae914926906058c69bd8ed5d",
            "fb9e3c1fa2034823b99681277f41b0a8",
            "cf0b06cab7b14500a705576c2ea5f82c",
            "ba83bf12fbf748d2b2413dd2a09ce43e",
            "fe31e7e2d361405f9a866404646d7d9f",
            "e45e5ebbb96c4d6b95ce8da95f2f8ee3",
            "1f75a225da9e4c61b90f499d3dcccf44",
            "b3057dcd2bad4bdca91bb0462c7f022e",
            "19a416b2255a459eb5e38ae16659573e",
            "05816b481eae4bf182fdbb7afb4c46e0",
            "2661f1212bf24795a3dd6aef1f0ef480",
            "24eb4d140be8444599a9a5deceb2c795",
            "57965686944149148385c5a4d7b25f7b",
            "ac9ddc3b629c4cfd9361c837522459a0",
            "3b6dc7f60d164613b10c0cfd5f392b8e",
            "f327c96becb04d379c081cc93702154a",
            "44cb035df14f4244a09615f4df4a1ba7",
            "eece86f33c224f09854751079ee04bcd",
            "1783c6a2114f4fe1b18f217e86b99244",
            "3d5ce661c27648f2897ad6834f7d3b66",
            "0c267c7eb24f4c3da05c540507bd4e3f",
            "23c934ab5d704d279d76d56e831cde02",
            "489398d37a734f01b95d0d709984f9ca",
            "f50066ed61e54105b59ca0b321134b63"
          ]
        },
        "id": "v0iPv1GDGxgT",
        "outputId": "eb0f2f34-7daf-48a3-ce7d-8bb4fdfd189f"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "35c1a1e0fa5442cc9b070ac887fd40ab",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/749 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1714: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "772e95b70424423b80fbd0f1ffb7e300",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "cf0b06cab7b14500a705576c2ea5f82c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)/main/tokenizer.json:   0%|          | 0.00/1.84M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ac9ddc3b629c4cfd9361c837522459a0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/414 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
        "    model_id,\n",
        "    use_auth_token=hf_auth\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bNysQFtPoaj7"
      },
      "source": [
        "Now we're ready to initialize the HF pipeline. There are a few additional parameters that we must define here. Comments explaining these have been included in the code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "qAYXi8ayKusU"
      },
      "outputs": [],
      "source": [
        "generate_text = transformers.pipeline(\n",
        "    model=model, tokenizer=tokenizer,\n",
        "    return_full_text=True,  # langchain expects the full text\n",
        "    task='text-generation',\n",
        "    # we pass model parameters here too\n",
        "    temperature=0.0,  # 'randomness' of outputs, 0.0 is the min and 1.0 the max\n",
        "    max_new_tokens=512,  # mex number of tokens to generate in the output\n",
        "    repetition_penalty=1.1  # without this output begins repeating\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8DG1WNTnJF1o"
      },
      "source": [
        "Confirm this is working:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lhFgmMr0JHUF",
        "outputId": "6b2c7bb1-db8c-4e0f-a6ab-e0f224a79bf1"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1270: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Explain to me the difference between nuclear fission and fusion.\n",
            "Nuclear fission is a process in which an atomic nucleus splits into two or more smaller nuclei, releasing a large amount of energy in the process. This occurs when an atom's nucleus is bombarded with a high-energy particle, such as a neutron. The resulting nuclei are typically smaller and lighter than the original nucleus, and the excess energy is released as radiation. Fission is the process used in nuclear power plants to generate electricity.\n",
            "Nuclear fusion, on the other hand, is the process by which two or more atomic nuclei combine to form a single, heavier nucleus. This process also releases a large amount of energy, but it requires the nuclei to be brought together at extremely high temperatures and pressures, typically found in the core of stars. Fusion is the process that powers the sun and other stars.\n",
            "The key difference between fission and fusion is the direction of the energy release. In fission, the energy is released outward from the nucleus, while in fusion, the energy is released inward, binding the nuclei together. Additionally, fission typically involves the splitting of heavy elements, while fusion involves the combining of light elements.\n"
          ]
        }
      ],
      "source": [
        "res = generate_text(\"Explain to me the difference between nuclear fission and fusion.\")\n",
        "print(res[0][\"generated_text\"])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0N3W3cj3Re1K"
      },
      "source": [
        "Now to implement this in LangChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "-8RxQYwHRg0N"
      },
      "outputs": [],
      "source": [
        "from langchain.llms import HuggingFacePipeline\n",
        "\n",
        "llm = HuggingFacePipeline(pipeline=generate_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 178
        },
        "id": "aiW0_FoQWG6J",
        "outputId": "05b4b748-dea8-464b-ee3a-e2aea84a4ae4"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"\\nNuclear fission is a process in which an atomic nucleus splits into two or more smaller nuclei, releasing a large amount of energy in the process. This occurs when an atom's nucleus is bombarded with a high-energy particle, such as a neutron. The resulting nuclei are typically smaller and lighter than the original nucleus, and the excess energy is released as radiation. Fission is the process used in nuclear power plants to generate electricity.\\nNuclear fusion, on the other hand, is the process by which two or more atomic nuclei combine to form a single, heavier nucleus. This process also releases a large amount of energy, but it requires the nuclei to be brought together at extremely high temperatures and pressures, typically found in the core of stars. Fusion is the process that powers the sun and other stars.\\nThe key difference between fission and fusion is the direction of the energy release. In fission, the energy is released outward from the nucleus, while in fusion, the energy is released inward, binding the nuclei together. Additionally, fission typically involves the splitting of heavy elements, while fusion involves the combining of light elements.\""
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "llm(prompt=\"Explain to me the difference between nuclear fission and fusion.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "5tv0KxJLvsIa"
      },
      "source": [
        "We still get the same output as we're not really doing anything differently here, but we have now added **Llama 2 70B Chat** to the LangChain library. Using this we can now begin using LangChain's advanced agent tooling, chains, etc, with **Llama 2**."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UjTR6XL54fc8"
      },
      "source": [
        "## Initializing an Conversational Agent\n",
        "\n",
        "Getting a conversational agent to work with open source models is incredibly hard. However, with Llama 2 70B it is now possible. Let's see how we can get it running!\n",
        "\n",
        "We first need to initialize the agent. Conversational agents require several things such as conversational `memory`, access to `tools`, and an `llm` (which we have already initialized)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "hmfX0_usXTx3"
      },
      "outputs": [],
      "source": [
        "from langchain.memory import ConversationBufferWindowMemory\n",
        "from langchain.agents import load_tools\n",
        "\n",
        "memory = ConversationBufferWindowMemory(\n",
        "    memory_key=\"chat_history\", k=5, return_messages=True, output_key=\"output\"\n",
        ")\n",
        "tools = load_tools([\"llm-math\"], llm=llm)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h0jhbRMw48ep"
      },
      "outputs": [],
      "source": [
        "from langchain.agents import AgentOutputParser\n",
        "from langchain.agents.conversational_chat.prompt import FORMAT_INSTRUCTIONS\n",
        "from langchain.output_parsers.json import parse_json_markdown\n",
        "from langchain.schema import AgentAction, AgentFinish\n",
        "\n",
        "class OutputParser(AgentOutputParser):\n",
        "    def get_format_instructions(self) -> str:\n",
        "        return FORMAT_INSTRUCTIONS\n",
        "\n",
        "    def parse(self, text: str) -> AgentAction | AgentFinish:\n",
        "        try:\n",
        "            # this will work IF the text is a valid JSON with action and action_input\n",
        "            response = parse_json_markdown(text)\n",
        "            action, action_input = response[\"action\"], response[\"action_input\"]\n",
        "            if action == \"Final Answer\":\n",
        "                # this means the agent is finished so we call AgentFinish\n",
        "                return AgentFinish({\"output\": action_input}, text)\n",
        "            else:\n",
        "                # otherwise the agent wants to use an action, so we call AgentAction\n",
        "                return AgentAction(action, action_input, text)\n",
        "        except Exception:\n",
        "            # sometimes the agent will return a string that is not a valid JSON\n",
        "            # often this happens when the agent is finished\n",
        "            # so we just return the text as the output\n",
        "            return AgentFinish({\"output\": text}, text)\n",
        "\n",
        "    @property\n",
        "    def _type(self) -> str:\n",
        "        return \"conversational_chat\"\n",
        "\n",
        "# initialize output parser for agent\n",
        "parser = OutputParser()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "CPwa9SWP4-dR"
      },
      "outputs": [],
      "source": [
        "from langchain.agents import initialize_agent\n",
        "\n",
        "# initialize agent\n",
        "agent = initialize_agent(\n",
        "    agent=\"chat-conversational-react-description\",\n",
        "    tools=tools,\n",
        "    llm=llm,\n",
        "    verbose=True,\n",
        "    early_stopping_method=\"generate\",\n",
        "    memory=memory,\n",
        "    #agent_kwargs={\"output_parser\": parser}\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Re7e6Ir361xV",
        "outputId": "a0acc2e3-4fec-49c7-fd7b-c962a2ff5e46"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "ChatPromptTemplate(input_variables=['input', 'chat_history', 'agent_scratchpad'], output_parser=None, partial_variables={}, messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], output_parser=None, partial_variables={}, template='Assistant is a large language model trained by OpenAI.\\n\\nAssistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\\n\\nAssistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.\\n\\nOverall, Assistant is a powerful system that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.', template_format='f-string', validate_template=True), additional_kwargs={}), MessagesPlaceholder(variable_name='chat_history'), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], output_parser=None, partial_variables={}, template='TOOLS\\n------\\nAssistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:\\n\\n> Calculator: Useful for when you need to answer questions about math.\\n\\nRESPONSE FORMAT INSTRUCTIONS\\n----------------------------\\n\\nWhen responding to me, please output a response in one of two formats:\\n\\n**Option 1:**\\nUse this if you want the human to use a tool.\\nMarkdown code snippet formatted in the following schema:\\n\\n```json\\n{{\\n    \"action\": string, \\\\ The action to take. Must be one of Calculator\\n    \"action_input\": string \\\\ The input to the action\\n}}\\n```\\n\\n**Option #2:**\\nUse this if you want to respond directly to the human. Markdown code snippet formatted in the following schema:\\n\\n```json\\n{{\\n    \"action\": \"Final Answer\",\\n    \"action_input\": string \\\\ You should put what you want to return to use here\\n}}\\n```\\n\\nUSER\\'S INPUT\\n--------------------\\nHere is the user\\'s input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):\\n\\n{input}', template_format='f-string', validate_template=True), additional_kwargs={}), MessagesPlaceholder(variable_name='agent_scratchpad')])"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "agent.agent.llm_chain.prompt"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "CjFtnFTcOH9_"
      },
      "source": [
        "We need to add special tokens used to signify the beginning and end of instructions, and the beginning and end of system messages. These are described in the Llama-2 model cards on Hugging Face."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "Mt8Ah5cRMBqN"
      },
      "outputs": [],
      "source": [
        "B_INST, E_INST = \"[INST]\", \"[/INST]\"\n",
        "B_SYS, E_SYS = \"<<SYS>>\\n\", \"\\n<</SYS>>\\n\\n\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "VzrpXdVA8K9r"
      },
      "outputs": [],
      "source": [
        "sys_msg = B_SYS + \"\"\"Assistant is a expert JSON builder designed to assist with a wide range of tasks.\n",
        "\n",
        "Assistant is able to respond to the User and use tools using JSON strings that contain \"action\" and \"action_input\" parameters.\n",
        "\n",
        "All of Assistant's communication is performed using this JSON format.\n",
        "\n",
        "Assistant can also use tools by responding to the user with tool use instructions in the same \"action\" and \"action_input\" JSON format. Tools available to Assistant are:\n",
        "\n",
        "- \"Calculator\": Useful for when you need to answer questions about math.\n",
        "  - To use the calculator tool, Assistant should write like so:\n",
        "    ```json\n",
        "    {{\"action\": \"Calculator\",\n",
        "      \"action_input\": \"sqrt(4)\"}}\n",
        "    ```\n",
        "\n",
        "Here are some previous conversations between the Assistant and User:\n",
        "\n",
        "User: Hey how are you today?\n",
        "Assistant: ```json\n",
        "{{\"action\": \"Final Answer\",\n",
        " \"action_input\": \"I'm good thanks, how are you?\"}}\n",
        "```\n",
        "User: I'm great, what is the square root of 4?\n",
        "Assistant: ```json\n",
        "{{\"action\": \"Calculator\",\n",
        " \"action_input\": \"sqrt(4)\"}}\n",
        "```\n",
        "User: 2.0\n",
        "Assistant: ```json\n",
        "{{\"action\": \"Final Answer\",\n",
        " \"action_input\": \"It looks like the answer is 2!\"}}\n",
        "```\n",
        "User: Thanks could you tell me what 4 to the power of 2 is?\n",
        "Assistant: ```json\n",
        "{{\"action\": \"Calculator\",\n",
        " \"action_input\": \"4**2\"}}\n",
        "```\n",
        "User: 16.0\n",
        "Assistant: ```json\n",
        "{{\"action\": \"Final Answer\",\n",
        " \"action_input\": \"It looks like the answer is 16!\"}}\n",
        "```\n",
        "\n",
        "Here is the latest conversation between Assistant and User.\"\"\" + E_SYS\n",
        "new_prompt = agent.agent.create_prompt(\n",
        "    system_message=sys_msg,\n",
        "    tools=tools\n",
        ")\n",
        "agent.agent.llm_chain.prompt = new_prompt"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "FT59fuKJVEvB"
      },
      "source": [
        "In the Llama 2 paper they mentioned that it was difficult to keep Llama 2 chat models following instructions over multiple interactions. One way they found that improves this is by inserting a reminder of the instructions to each user query. We do that here:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "s9HSoPJknbmI"
      },
      "outputs": [],
      "source": [
        "instruction = B_INST + \" Respond to the following in JSON with 'action' and 'action_input' values \" + E_INST\n",
        "human_msg = instruction + \"\\nUser: {input}\"\n",
        "\n",
        "agent.agent.llm_chain.prompt.messages[2].prompt.template = human_msg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KEgrOtXHEnPy",
        "outputId": "ea310ad4-3d4d-4447-eef3-4511f3772548"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "ChatPromptTemplate(input_variables=['input', 'chat_history', 'agent_scratchpad'], output_parser=None, partial_variables={}, messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], output_parser=None, partial_variables={}, template='<<SYS>>\\nAssistant is a expert JSON builder designed to assist with a wide range of tasks.\\n\\nAssistant is able to respond to the User and use tools using JSON strings that contain \"action\" and \"action_input\" parameters.\\n\\nAll of Assistant\\'s communication is performed using this JSON format.\\n\\nAssistant can also use tools by responding to the user with tool use instructions in the same \"action\" and \"action_input\" JSON format. Tools available to Assistant are:\\n\\n- \"Calculator\": Useful for when you need to answer questions about math.\\n  - To use the calculator tool, Assistant should write like so:\\n    ```json\\n    {{\"action\": \"Calculator\",\\n      \"action_input\": \"sqrt(4)\"}}\\n    ```\\n\\nHere are some previous conversations between the Assistant and User:\\n\\nUser: Hey how are you today?\\nAssistant: ```json\\n{{\"action\": \"Final Answer\",\\n \"action_input\": \"I\\'m good thanks, how are you?\"}}\\n```\\nUser: I\\'m great, what is the square root of 4?\\nAssistant: ```json\\n{{\"action\": \"Calculator\",\\n \"action_input\": \"sqrt(4)\"}}\\n```\\nUser: 2.0\\nAssistant: ```json\\n{{\"action\": \"Final Answer\",\\n \"action_input\": \"It looks like the answer is 2!\"}}\\n```\\nUser: Thanks could you tell me what 4 to the power of 2 is?\\nAssistant: ```json\\n{{\"action\": \"Calculator\",\\n \"action_input\": \"4**2\"}}\\n```\\nUser: 16.0\\nAssistant: ```json\\n{{\"action\": \"Final Answer\",\\n \"action_input\": \"It looks like the answer is 16!\"}}\\n```\\n\\nHere is the latest conversation between Assistant and User.\\n<</SYS>>\\n\\n', template_format='f-string', validate_template=True), additional_kwargs={}), MessagesPlaceholder(variable_name='chat_history'), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], output_parser=None, partial_variables={}, template=\"[INST] Respond to the following in JSON with 'action' and 'action_input' values [/INST]\\nUser: {input}\", template_format='f-string', validate_template=True), additional_kwargs={}), MessagesPlaceholder(variable_name='agent_scratchpad')])"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "agent.agent.llm_chain.prompt"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_LDoIUbyVbCG"
      },
      "source": [
        "Now we can begin asking questions..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hjnM44q76itO",
        "outputId": "26cc3c7f-74f1-46ae-a08f-3a1022146f0d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
            "\u001b[32;1m\u001b[1;3m\n",
            "Assistant: ```json\n",
            "{\"action\": \"Final Answer\",\n",
            " \"action_input\": \"I'm good thanks, how are you?\"}\n",
            "```\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'input': 'hey how are you today?',\n",
              " 'chat_history': [],\n",
              " 'output': \"I'm good thanks, how are you?\"}"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "agent(\"hey how are you today?\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "adBWZl1v-iwA",
        "outputId": "37ad266b-8b5b-4c9b-80e8-6d921c8b4b01"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
            "\u001b[32;1m\u001b[1;3m\n",
            "Assistant: ```json\n",
            "{\"action\": \"Calculator\",\n",
            " \"action_input\": \"4**2.1\"}\n",
            "```\u001b[0m\n",
            "Observation: \u001b[36;1m\u001b[1;3mAnswer: 18.37917367995256\u001b[0m\n",
            "Thought:\u001b[32;1m\u001b[1;3m\n",
            "\n",
            "AI: \n",
            "Assistant: ```json\n",
            "{\"action\": \"Final Answer\",\n",
            " \"action_input\": \"It looks like the answer is 18.37917367995256!\"}\n",
            "```\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'input': 'what is 4 to the power of 2.1?',\n",
              " 'chat_history': [HumanMessage(content='hey how are you today?', additional_kwargs={}, example=False),\n",
              "  AIMessage(content=\"I'm good thanks, how are you?\", additional_kwargs={}, example=False)],\n",
              " 'output': 'It looks like the answer is 18.37917367995256!'}"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "agent(\"what is 4 to the power of 2.1?\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E3qWzs1M-sqp",
        "outputId": "5a35464c-0ce3-439c-8a27-4447dfc5ad07"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
            "\u001b[32;1m\u001b[1;3m\n",
            "Assistant: ```json\n",
            "{\"action\": \"Calculator\",\n",
            " \"action_input\": \"18.37917367995256 * 3\"}\n",
            "```\u001b[0m\n",
            "Observation: \u001b[36;1m\u001b[1;3mAnswer: 55.13752103985769\u001b[0m\n",
            "Thought:"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1083: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32;1m\u001b[1;3m\n",
            "\n",
            "AI: \n",
            "Assistant: ```json\n",
            "{\"action\": \"Final Answer\",\n",
            " \"action_input\": \"It looks like the answer is 55.13752103985769!\"}\n",
            "```\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{'input': 'can you multiply that by 3?',\n",
              " 'chat_history': [HumanMessage(content='hey how are you today?', additional_kwargs={}, example=False),\n",
              "  AIMessage(content=\"I'm good thanks, how are you?\", additional_kwargs={}, example=False),\n",
              "  HumanMessage(content='what is 4 to the power of 2.1?', additional_kwargs={}, example=False),\n",
              "  AIMessage(content='It looks like the answer is 18.37917367995256!', additional_kwargs={}, example=False)],\n",
              " 'output': 'It looks like the answer is 55.13752103985769!'}"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "agent(\"can you multiply that by 3?\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "kkO9FR7VUrZS"
      },
      "source": [
        "With that we have our **open source** conversational agent running on Colab with ~38GB of RAM.\n",
        "\n",
        "---"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "A100",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}